Course: Python Training
Topic: Introduction to Web Scraping Using Beautiful Soup
Course Link: Python Training Course at JustAcademy
Demo Link: Register for a Free Course Demo
Website: Visit JustAcademy
Reference Documentation: Official Beautiful Soup Documentation
Web scraping is the process of collecting specific information from web pages and converting it into a structured format that can be analyzed, stored, or used in applications. Websites commonly display information using HTML, which contains elements such as headings, paragraphs, links, tables, lists, and images.
Python provides several libraries for web scraping. Beautiful Soup is one of the most beginner-friendly tools because it allows developers to search, navigate, and extract information from HTML and XML documents using simple Python methods.
For example, suppose a website contains product names, prices, descriptions, and ratings. Instead of copying each item manually, a Python program can retrieve the permitted webpage content, parse its HTML, and extract the required fields automatically.
Beautiful Soup is a Python library used to parse HTML and XML documents. It creates a navigable structure from the supplied markup and provides methods for finding elements, accessing attributes, extracting text, and traversing the document tree.
Beautiful Soup does not download webpages by itself and does not execute JavaScript. A separate HTTP client, such as the Requests library, is commonly used to retrieve a webpage before Beautiful Soup processes its HTML.
A basic scraping program usually follows these steps:
Website
|
v
HTTP Request using Requests
|
v
HTML Response
|
v
Beautiful Soup Parser
|
v
Find Required HTML Elements
|
v
Extract and Clean Data
|
v
Store or Analyze the ResultsTo begin, install Beautiful Soup and Requests in your Python environment. Requests retrieves webpage content, while Beautiful Soup parses that content.
python -m pip install beautifulsoup4 requestsThe installed package is named beautifulsoup4, but its Python import name is bs4.
import requests
from bs4 import BeautifulSoup
print("Requests and Beautiful Soup imported successfully.")If the program runs without an import error, both libraries are available in the active Python environment.
Python includes the built-in html.parser parser. You can also install lxml, which is often useful when parsing performance matters.
python -m pip install lxmlHTML describes the structure of a webpage through elements, tags, attributes, and text content. Understanding this structure helps you identify the correct elements to extract.
<!DOCTYPE html>
<html>
<head>
<title>Python Learning Resources</title>
</head>
<body>
<h1 id="page-title">Learn Python</h1>
<p class="description">Explore Python tutorials.</p>
<a href="https://example.com/tutorial">Open Tutorial</a>
</body>
</html>| Component | Example | Purpose |
|---|---|---|
| Tag | <h1> |
Defines a type of HTML element. |
| Attribute | id="page-title" |
Provides additional information about an element. |
| Class | class="description" |
Groups elements for styling or selection. |
| Text | Learn Python |
Content displayed inside an element. |
| Link destination | href="https://example.com/tutorial" |
Specifies where a link points. |
The BeautifulSoup class accepts HTML markup and a parser name. It creates an object that you can search and navigate.
from bs4 import BeautifulSoup
html = """
<html>
<head>
<title>Python Learning Resources</title>
</head>
<body>
<h1>Learn Python</h1>
<p>Explore Python tutorials.</p>
</body>
</html>
"""
soup = BeautifulSoup(html, "html.parser")
print(soup.title)
print(soup.h1)
print(soup.p)Expected output:
<title>Python Learning Resources</title>
<h1>Learn Python</h1>
<p>Explore Python tutorials.</p>The returned objects represent HTML tags. Use .get_text() or the .text property to extract their text content.
print(soup.title.get_text())
print(soup.h1.get_text())
print(soup.p.get_text())Expected output:
Python Learning Resources
Learn Python
Explore Python tutorials.Beautiful Soup exposes several types of objects that represent different parts of a document.
| Object | Description | Example |
|---|---|---|
BeautifulSoup |
Represents the parsed document. | soup |
Tag |
Represents an HTML or XML element. | soup.h1 |
NavigableString |
Represents text inside a tag. | soup.p.string |
Comment |
Represents an HTML comment. | A comment inside the document tree. |
Two of the most frequently used methods are find() and find_all().
The find() method returns the first matching element, or None if no match is found.
from bs4 import BeautifulSoup
html = """
<h2>First Topic</h2>
<h2>Second Topic</h2>
"""
soup = BeautifulSoup(html, "html.parser")
heading = soup.find("h2")
if heading:
print(heading.get_text(strip=True))Output:
First TopicThe find_all() method returns a collection of all matching elements.
headings = soup.find_all("h2")
for heading in headings:
print(heading.get_text(strip=True))Output:
First Topic
Second Topic| Feature | find() | find_all() |
|---|---|---|
| Results | First matching element | All matching elements |
| Return type | Tag or None | List-like ResultSet |
| Typical use | Page title or main heading | Multiple articles or product cards |
| Iteration | Not normally required for one result | Commonly used in a loop |
HTML classes and IDs help identify specific elements. In Beautiful Soup, use class_ to search by class because class is a reserved Python keyword.
html = """
<p class="summary">First summary</p>
<p class="summary">Second summary</p>
<p class="other">Other content</p>
"""
soup = BeautifulSoup(html, "html.parser")
summaries = soup.find_all("p", class_="summary")
for item in summaries:
print(item.get_text(strip=True))html = """
<h1 id="main-heading">Python Web Scraping</h1>
"""
soup = BeautifulSoup(html, "html.parser")
heading = soup.find(id="main-heading")
if heading:
print(heading.get_text(strip=True))Output:
Python Web ScrapingBeautiful Soup supports many CSS selectors through select() and select_one(). CSS selectors can be useful when elements are identified by a combination of tags, classes, IDs, and relationships.
| Selector | Meaning |
|---|---|
p |
Select all paragraph elements. |
.product |
Select elements with the product class. |
#main |
Select the element with the main ID. |
div.product |
Select div elements with the product class. |
article h2 |
Select h2 elements inside article elements. |
a[href] |
Select links that have an href attribute. |
html = """
<div class="product">
<h2>Keyboard</h2>
</div>
<div class="product">
<h2>Mouse</h2>
</div>
"""
soup = BeautifulSoup(html, "html.parser")
for item in soup.select(".product h2"):
print(item.get_text(strip=True))Output:
Keyboard
MouseUse select_one() when only the first matching element is required.
first_product = soup.select_one(".product h2")
if first_product:
print(first_product.get_text(strip=True))Links are generally represented by the HTML <a> tag. The link text is stored inside the element, while the destination is commonly stored in the href attribute.
html = """
<a href="https://example.com/python">Python Guide</a>
<a href="https://example.com/data">Data Science Guide</a>
"""
soup = BeautifulSoup(html, "html.parser")
for link in soup.find_all("a", href=True):
text = link.get_text(strip=True)
url = link.get("href")
print(text, "-", url)Output:
Python Guide - https://example.com/python
Data Science Guide - https://example.com/dataWhen extracting links from a real webpage, some destinations may be relative paths such as /articles/python. Use Python's urllib.parse.urljoin() to resolve those paths against the page's base URL.
from urllib.parse import urljoin
base_url = "https://example.com/resources/"
relative_url = "../python-guide"
absolute_url = urljoin(base_url, relative_url)
print(absolute_url)Output:
https://example.com/python-guideImage elements commonly use the <img> tag. Useful attributes include src, which indicates the image source, and alt, which provides alternative text.
html = """
<img src="/images/python.png" alt="Python logo">
"""
soup = BeautifulSoup(html, "html.parser")
image = soup.find("img")
if image:
print("Source:", image.get("src"))
print("Alternative text:", image.get("alt"))Output:
Source: /images/python.png
Alternative text: Python logoRemember that a relative image source is not automatically converted into a complete URL. Resolve it using urljoin() when needed.
Some websites organize information into HTML tables or lists. Beautiful Soup can locate rows, cells, and list items and extract their text.
html = """
<table>
<tr><th>Product</th><th>Price</th></tr>
<tr><td>Keyboard</td><td>1200</td></tr>
<tr><td>Mouse</td><td>600</td></tr>
</table>
"""
soup = BeautifulSoup(html, "html.parser")
for row in soup.select("table tr"):
cells = [
cell.get_text(strip=True)
for cell in row.find_all(["th", "td"])
]
print(cells)Output:
['Product', 'Price']
['Keyboard', '1200']
['Mouse', '600']html = """
<ul>
<li>Python</li>
<li>Beautiful Soup</li>
<li>Requests</li>
</ul>
"""
soup = BeautifulSoup(html, "html.parser")
items = soup.find_all("li")
for item in items:
print(item.get_text(strip=True))This section combines Requests with Beautiful Soup. The example uses example.com, a reserved example domain, to demonstrate the workflow without targeting a third-party content website.
import requests
from bs4 import BeautifulSoup
url = "https://example.com"
try:
response = requests.get(
url,
timeout=10,
headers={"User-Agent": "LearningScraper/1.0"}
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
title = soup.find("title")
if title:
print("Page title:", title.get_text(strip=True))
else:
print("No title found.")
except requests.exceptions.RequestException as error:
print("Request failed:", error)requests.get() sends an HTTP GET request.timeout=10 prevents the request from waiting indefinitely.raise_for_status() raises an exception for unsuccessful HTTP status codes.response.text provides the response body as decoded text.BeautifulSoup() parses the returned HTML.find("title") locates the page title.The actual content depends on the response returned by the server. A website may return an error page, redirect the request, restrict automated access, or provide different markup.
The following example demonstrates how to extract article headings and links from HTML. It uses a sample HTML document so the result is predictable and does not depend on an external website.
from bs4 import BeautifulSoup
from urllib.parse import urljoin
base_url = "https://example.com/news/"
html = """
<html>
<body>
<article>
<h2>
<a href="/news/python-basics">Python Basics</a>
</h2>
</article>
<article>
<h2>
<a href="/news/web-scraping">Web Scraping Introduction</a>
</h2>
</article>
</body>
</html>
"""
soup = BeautifulSoup(html, "html.parser")
articles = soup.select("article h2 a")
for article in articles:
title = article.get_text(strip=True)
href = article.get("href")
if not href:
continue
full_url = urljoin(base_url, href)
print("Title:", title)
print("URL:", full_url)
print("-" * 30)Expected output:
Title: Python Basics
URL: https://example.com/news/python-basics
------------------------------
Title: Web Scraping Introduction
URL: https://example.com/news/web-scraping
------------------------------After extracting information, you may want to store it in a CSV file for analysis in spreadsheet software or Python data-analysis libraries.
import csv
from bs4 import BeautifulSoup
html = """
<article>
<h2>Python Basics</h2>
<p class="summary">Learn Python fundamentals.</p>
</article>
<article>
<h2>Web Scraping</h2>
<p class="summary">Extract structured webpage data.</p>
</article>
"""
soup = BeautifulSoup(html, "html.parser")
with open(
"articles.csv",
"w",
newline="",
encoding="utf-8"
) as file:
writer = csv.writer(file)
writer.writerow(["Title", "Summary"])
for article in soup.select("article"):
title_tag = article.find("h2")
summary_tag = article.select_one(".summary")
title = (
title_tag.get_text(strip=True)
if title_tag else ""
)
summary = (
summary_tag.get_text(" ", strip=True)
if summary_tag else ""
)
writer.writerow([title, summary])
print("Data saved to articles.csv")This example extracts data from sample markup and writes it to articles.csv. For live scraping, ensure you have permission to retrieve the page content and validate the extracted fields before saving.
Webpage structures are not always consistent. An expected tag may be missing, a class name may change, or an element may contain extra whitespace. Robust scraping code should handle these possibilities.
heading = soup.find("h1")
if heading is not None:
title = heading.get_text(" ", strip=True)
else:
title = "Title unavailable"
print(title)html = """
<p>
Learn Python
<strong>step by step</strong>.
</p>
"""
soup = BeautifulSoup(html, "html.parser")
paragraph = soup.find("p")
if paragraph:
text = paragraph.get_text(" ", strip=True)
print(text)Output:
Learn Python step by step.The separator passed to get_text() helps keep words separated when text comes from nested HTML elements.
Beautiful Soup relies on a parser to interpret HTML. Different parsers may construct slightly different document trees when the original markup is malformed.
| Parser | Characteristics | Typical Use |
|---|---|---|
html.parser |
Built into Python; no additional parser package required. | Beginner projects and general parsing. |
lxml |
Often fast; requires installation of an external package. | Projects where parsing speed matters. |
html5lib |
Designed to handle HTML similarly to modern browsers; generally slower. | Documents requiring browser-like HTML parsing behavior. |
Example using the built-in parser:
soup = BeautifulSoup(html, "html.parser")Example using lxml after installation:
soup = BeautifulSoup(html, "lxml")If a page's required data is available only after JavaScript execution, consider an official API or an appropriate browser-automation tool when its use is permitted. Beautiful Soup can still parse HTML obtained from such a workflow, but it does not perform the browser execution itself.
| Problem | Possible Cause | Solution |
|---|---|---|
ModuleNotFoundError: No module named 'bs4' |
Beautiful Soup is missing from the active environment. | Run python -m pip install beautifulsoup4 using the Python environment running your script. |
find() returns None |
The element or selector does not match the supplied HTML. | Inspect the HTML and verify the tag, class, ID, and attributes. |
| Empty extraction results | The content may be dynamically rendered or the returned response differs from the expected page. | Inspect the response status and HTML; check whether an official API is available. |
| HTTP error | The server returned an unsuccessful status code. | Use raise_for_status(), handle exceptions, and review permitted access requirements. |
| Incorrect links | Extracted destinations are relative URLs. | Resolve them with urljoin(). |
| Unexpected text spacing | Text is nested inside multiple HTML elements. | Use get_text(" ", strip=True) and inspect the result. |
| Different output on another system | Different parsers or package environments may be in use. | Specify the parser explicitly and keep dependencies consistent. |
Web scraping should be performed responsibly. Publicly accessible information is not automatically free of usage restrictions, and technical accessibility does not imply permission to collect or reuse every kind of data.
robots.txt guidance as one input to your access decisions; it is not a substitute for legal permission or terms of service.article_title and product_price.get_text(strip=True) or an appropriate separator to normalize extracted text.Create a sample HTML document containing three <h2> elements. Use Beautiful Soup to extract and print all headings.
Create multiple paragraph elements with different classes. Extract only the paragraphs with the class description.
Create a sample document with five anchor elements. Print each link's text and href attribute.
Create an HTML table containing product names and prices. Extract each row and convert it into a list of values.
Extract article titles and summaries from sample HTML and save the results to a CSV file using Python's csv module.
Using a website or dataset that explicitly permits automated access, retrieve its HTML with Requests, parse it with Beautiful Soup, extract a few specified fields, handle missing elements, and save the validated output. Use a timeout and avoid sending excessive requests.
Project Objective: Build a small program that extracts titles and URLs from a sample learning-resource webpage.
html = """
<div class="resource">
<h3>Python Basics</h3>
<a href="/learn/python">Read More</a>
</div>
<div class="resource">
<h3>Web Scraping</h3>
<a href="/learn/scraping">Read More</a>
</div>
"""from bs4 import BeautifulSoup
from urllib.parse import urljoin
base_url = "https://example.com"
soup = BeautifulSoup(html, "html.parser")
resources = []
for item in soup.select(".resource"):
title_tag = item.find("h3")
link_tag = item.find("a", href=True)
title = (
title_tag.get_text(strip=True)
if title_tag else ""
)
url = (
urljoin(base_url, link_tag["href"])
if link_tag else ""
)
if title and url:
resources.append({
"title": title,
"url": url
})
for resource in resources:
print(resource["title"], "-", resource["url"])Expected output:
Python Basics - https://example.com/learn/python
Web Scraping - https://example.com/learn/scrapingAnswer: Web scraping is the automated process of retrieving permitted webpage content and extracting selected information into a structured form.
Answer: Beautiful Soup is a Python library that parses HTML or XML and provides methods for searching, navigating, and extracting information from the parsed document.
Answer: No. Beautiful Soup parses supplied markup. Libraries such as Requests are commonly used to retrieve webpage content.
Answer: find() returns the first matching element or None. find_all() returns all matching elements in a list-like result.
Answer: Use tag.get_text(strip=True) or tag.text. The get_text() method offers more control over whitespace and separators.
Answer: Use soup.find_all(class_="description") or a CSS selector such as soup.select(".description").
Answer: Locate an anchor element and retrieve its href attribute using link.get("href").
Answer: An HTML parser interprets markup and builds a structured representation of the document that can be searched and navigated.
Answer: It returns None. Check for this result before calling methods on the returned value.
Answer: Beautiful Soup does not execute JavaScript. If the required data is not present in the returned HTML, consider an authorized API or another permitted method for obtaining the rendered content.
Answer: Requests handles HTTP communication and retrieves the response. Beautiful Soup parses the returned markup and extracts the required elements.
Answer: Use timeouts, handle HTTP errors, specify a parser, choose suitable selectors, check for missing elements, validate results, and follow the target website's access requirements.
beautifulsoup4 and requests.BeautifulSoup from bs4.BeautifulSoup(html, "html.parser").find() for the first match and find_all() for multiple matches.get_text() and attributes with get().urljoin().Beautiful Soup is a practical starting point for learning web scraping with Python. It makes HTML and XML documents easier to navigate and provides convenient methods for extracting text, links, images, tables, and other elements. When combined with Requests, it supports a simple workflow for retrieving and parsing webpage content.
Begin by practicing on sample HTML, learn to identify elements through tags and attributes, and then build small projects using sources that permit automated access. Reliable scrapers combine correct parsing with error handling, data validation, responsible request behavior, and respect for website policies.
To continue learning Python, visit the JustAcademy Python Training Course, register for a course demo, or explore the JustAcademy website.
